Learning Convolutional Action Primitives from Multimodal Timeseries Data*

Fine-grained action recognition is important for many applications such as human-robot interaction, automated skill assessment, and surveillance. The goal is to predict what action is occurring at any point in a timeseries sequence. While recent work has improved recognition performance in robotics applications, these methods often require hand-crafted features or large use of domain knowledge. Furthermore these methods tend to model actions using pointwise estimates of individual frames or statistics on collections of frames. In this work we develop a notion of an action primitive that models how generic features transition over the course of an action. Our Latent Convolutional Skip Chain Conditional Random Field (LC-SCCRF) model learns a set of interpretable and composable action primitives. We apply our model to the cooking and robotic surgery domains using the University of Dundee 50 Salads dataset and the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS). Each consists of multimodal timeseries data including video, robot kinematics, and/or kitchen object accelerations. Our recognition performance on 50 Salads and JIGSAWS are 18.0% and 5.3% higher than the state of the art. Our model has the advantage of being more general than many recent approaches and performs well without requiring hand-crafted features or intricate domain knowledge. Upon publication we will release our LC-SC-CRF code and the features used on the two datasets.